Allow truncation when embedding #14493
Conversation
|
@ngxson Please check |
There was a problem hiding this comment.
This can be an intermediate solution for now, but I think the whole embedding system need to be reworked at some point.
Currently, we don't have to process the whole input in one single batch if we're using last pooling with causal attention. Having this can be useful for newer models like Qwen, while also unlock using one single model for both text generation & embeddings. @ggerganov WDYT?
| // Note: If the input was truncated (slot.truncated == true), this embedding | ||
| // represents only the processed portion of the original input | ||
| for (int i = 0; i < batch.n_tokens; ++i) { |
There was a problem hiding this comment.
An easier way to avoid touching this loop - which now becomes more and more fragile - is to simply handle this truncation in launch_slot_with_task. You can truncate slot.prompt_tokens right after it's being std::moved
|
I agree that the embeddings handling in This change specifically does not seems to improve the situation - we introduce another special case with extra parameter and branches. Better to implement this properly later. |
Sometimes it frustrates me because llama-server automatically stops when slot.n_ctx < input token length in embedding task. I want it to be able to truncate the input token, as an option.